Bagging and Boosting a Treebank Parser

Bagging and boosting, two effective machine learning techniques, are applied to natural language parsing. Experiments using these techniques with a trainable statistical parser are described. The best resulting system provides roughly as large of a gain in F-measure as doubling the corpus size. Error analysis of the result of the boosting technique reveals some inconsistent annotations in the Penn Treebank, suggesting a semi-automatic method for finding inconsistent treebank annotations. 1 I n t r o d u c t i o n Henderson and Brill (1999) showed that independent human research efforts produce parsers that can be combined for an overall boost in accuracy. Finding an ensemble of parsers designed to complement each other is clearly desirable. The parsers would need to be the result of a unified research effort, though, in which the errors made by one parser are targeted with priority by the developer of another parser. A set of five parsers which each achieve only 40% exact sentence accuracy would be extremely valuable if they made errors in such a way that at least two of the five were correct on any given sentence (and the others abstained or were wrong in different ways). 100% sentence accuracy could be achieved by selecting the hypothesis that was proposed by the two parsers that agreed completely. In this paper, the task of automatically creating complementary parsers is separated from the task of creating a single parser. This facilitates study of the ensemble creation techniques in isolation. The result is a method for increasing parsing performance by creating an ensemble of parsers, each produced from data using the same parser induction algorithm. 2 B a g g i n g and Pars ing 2.1 Background The work of Efron and Tibshirani (1993) enabled Breiman's refinement and application of their techniques for machine learning (Breiman, 1996). His technique is called bagging, short for "bootstrap aggregating". In brief, bootstrap techniques and bagging in particular reduce the systematic biases many estimation techniques introduce by aggregating estimates made from randomly drawn representative resamplings of those datasets. Bagging at tempts to find a set of classifiers which are consistent with the training data, different from each other, and distributed such that the aggregate sample distribution approaches the distribution of samples in the training set. Algorithm: Bagging Predictors (Breiman, 1996) (1) Given: training set • = { (y i ,x~) , i E { 1 . . . m } } drawn from the set A of possible training sets where Yi is the label for example x~, classification induction algorithm q2 : A --* • with classification algorithm C e • and ¢ : X ~ Y . 1. Create k bootstrap replicates o f / : by sampling m items from E with replacement. Call them L 1 . . . L k . 2. For each j e { 1 . . . k } , Let Cj = ~ ( £ j ) be the classifier induced using Lj as the training set. 3. If Y is a discrete set, then for each x~ observed in the test set, yi = m o d e (¢ j (x i ) . . . Cj(x~)). y~ is the value predicted by the most predictors, the majority vote. 2.2 Bagging for Parsing An algorithm that applies the technique of bagging to parsing is given in Algorithm 2. Previous work on combining independent parsers is leveraged to produce the combined parser. The rest of the algorithm is a straightforward transformation of bagging for classifiers. Exploratory work in this vein was described by HajiC et al. (1999). Algorithm: Bagging A Parser (2) Given: A corpus (again as a f u n c t i o n ) C : S × T ~ N, S is the set of possible sentences, and T is the set of trees, with size m = [C] = ~ s , t C(s, t) and parser induction algorithm g. 1. Draw k bootstrap replicates C1 . . . Ck of C each containing m samples of (s,t) pairs randomly